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The  aviation  industry  has  forecast  a  potential  66-percent  increase  in 
passenger  travel  from  1999  to  2008.  The  U.S.  aviation  accident  rate,  which 
has  remained  relatively  constant  over  the  past  two  decades,1  must  be 
substantially  lowered  to  avoid  escalating  numbers  of  aviation  deaths  as  air 
traffic  increases.  A  key  to  reducing  the  aviation  accident  rate  is  for  the 
Federal  Aviation  Administration  (faa)  to  have  an  effective  process  for 
inspecting  the  nation’s  airline  operations.  In  the  past,  we  and  others  have 
expressed  concerns  about  the  adequacy  of  faa’s  inspection  process  to 
meet  that  challenge.  Concerns  about  the  inspection  process  focused  on 
unstructured,  nonsystematic  inspections  that  produced  few  reports  of 
safety  problems  and  on  the  adequacy  of  inspectors’  technical  training. 
These  concerns  also  raised  questions  about  the  quality  and  consistency  of 
the  resulting  inspection  data  and  their  usefulness  for  conducting  analyses 
and  targeting  faa’s  resources  to  the  greatest  safety  risks. 

faa  has  responded  to  these  concerns  by  redesigning  the  safety  inspection 
system  that  it  uses  to  oversee  the  nation’s  airlines,  faa  began  using  the 
revised  approach,  called  the  Air  Transportation  Oversight  System  (atos), 
for  a  limited  number  of  airlines  during  the  system’s  initial  implementation 
on  October  1,  1998.  Currently,  the  nation’s  10  largest  passenger  airlines  are 
under  atos.2  At  your  request,  we  reviewed  faa’s  implementation  of  the  new 
system.  This  report  summarizes  our  work  by  addressing  the  following 
questions: 

•  To  what  extent  does  atos  address  past  concerns  about  faa’s  aviation 
safety  inspections? 

•  What  factors,  if  any,  surfaced  during  the  implementation  of  atos  that  could 
impede  its  success? 


lrThe  National  Transportation  Safety  Board’s  statistics  show  an  accident  rate  of  5  fatal  accidents  for 
each  10  million  flights  on  scheduled  and  nonscheduled  service  by  U.S.  airlines  operating  under  part 
121  of  the  Federal  Aviation  Regulations  from  1982  through  1998. 

2These  airlines  are  Alaska,  America  West,  American,  Continental,  Delta,  Northwest,  Southwest,  Trans 
World,  United,  and  US  Airways. 


Page  1 


GAO/RCED-99-183  FAA’s  New  Aviation  Inspection  System 


B-281502 


•  What  is  faa  doing  to  address  any  factors  that  could  impede  the  success  of 

ATOS? 


Results  in  Brief 


The  Air  Transportation  Oversight  System  is  largely  responsive  to  past 
concerns  raised  about  key  aspects  of  faa’s  aviation  safety  inspections  and 
the  usefulness  of  inspection  data.  These  concerns  centered  on  faa’s 
unstructured  inspection  process,  the  adequacy  of  technical  training  for 
inspectors,  the  quality  and  consistency  of  inspection  data,  and  the 
usefulness  of  those  data  for  identifying  safety  problems  and  targeting  the 
agency’s  resources  to  the  greatest  risks.  Addressing  these  concerns 
involved  a  fundamental  redesign  of  the  way  faa  inspects  the  nation’s 
airlines.  To  improve  inspection  quality,  the  new  program  emphasizes  a 
system  safety  approach  that  goes  beyond  spot-checking  airlines  for 
compliance  with  Federal  Aviation  Regulations.  Using  safety  principles 
originally  created  for  the  nuclear  industry,  it  calls  for  a  systematic  review 
of  airlines’  policies  and  procedures  to  ensure  that  they  incorporate  basic 
safety  principles,  such  as  clear  lines  of  responsibility  and  written 
documentation.  It  fosters  more  consistent,  structured  inspections  by 
standardizing  inspection  tasks,  linking  inspectors’  training  more  closely  to 
their  assigned  responsibilities,  and  using  teams  rather  than  individual 
inspectors  to  perform  many  inspections.  The  program  also  calls  for  a 
number  of  enhancements  to  improve  the  usefulness  of  inspection  data  for 
analysis  and  targeting.  They  include  a  standardized  database  for  reporting 
inspection  results  and  the  addition  of  data  quality  assurance  managers  and 
analysts.  The  goal  of  this  redesign  is  to  target  inspection  resources  to 
those  areas  that  present  the  greatest  safety  risks. 

atos  offers  promise  for  significantly  strengthening  faa’s  inspection 
process,  but  faa  must  also  address  the  problems  identified  in  this  report  to 
ensure  that  the  new  system  fulfills  its  promise,  faa’s  ability  to  conduct 
effective  inspections  remains  limited  by  a  lack  of  clear  guidance,  staff 
turnover,  and  continued  difficulties  with  the  adequacy  of  inspectors’ 
technical  training  and  experience.  The  anticipated  enhancements  to  make 
inspection  data  more  useful  have  not  been  achieved  because  of  problems 
with  reporting  requirements  and  the  incompatibility  of  the  program’s 
database  with  faa’s  primary  inspection  analysis  system.  In  addition,  faa 
planned  to  hire  an  analyst  for  each  of  its  new  inspection  teams  to  analyze 
inspection  data  for  safety  trends  and  to  guide  inspection  planning,  but  has 
not  yet  done  so  because  of  higher  priorities,  such  as  increasing  salaries  for 
air  traffic  controllers.  These  problems  resulted  largely  from  faa’s  decision 
to  implement  the  new  inspection  system  on  an  overly  ambitious  schedule. 
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Meeting  faa’s  target  date  for  implementation  meant  that  complex,  critical 
steps — such  as  developing  guidance,  training  inspectors,  creating 
databases,  and  consulting  with  the  affected  airlines — had  to  be 
compressed  into  a  very  short  time. 

faa  has  begun  to  address  some  of  the  problems  with  the  program,  faa 
management  officials  acknowledged  that  atos  faces  significant  challenges. 
They  agreed  with  our  conclusion  that  the  program  should  not  be  expanded 
beyond  the  nation’s  10  major  airlines  until  the  problems  that  emerged 
during  the  program’s  initial  implementation  are  resolved.  However,  some 
of  these  problems  have  not  yet  been  fully  addressed.  Consequently,  we 
recommend  several  specific  actions  to  clarify  the  program  guidance  and 
improve  the  usefulness  of  faa’s  database  for  targeting  inspection 
resources  to  the  areas  of  greatest  potential  safety  risk. 


Background 


Federal  law  establishes  that  the  safety  of  U.S.  air  passengers  is  a  joint 
responsibility  of  the  airlines  and  faa.  The  airlines  are  responsible  for 
operating  their  aircraft  safely.  FAA  is  responsible  for,  among  other  things, 
examining  an  airline’s  operations  when  the  airline  seeks  a  certificate  to 
operate  and  for  conducting  periodic  inspections  to  ensure  continued 
compliance  with  safety  regulations.  Within  faa,  the  Office  of  Flight 
Standards  Service  develops  the  Federal  Aviation  Regulations  that  airlines 
must  follow  and  prepares  guidance  on  how  faa’s  safety  inspectors  should 
perform  inspections. 

faa  has  nearly  3,300  safety  inspectors  located  in  101  district  offices 
throughout  the  United  States.  One  of  the  inspectors’  primary  functions  is 
conducting  what  FAA  calls  “routine  surveillance” — a  process  of  continuous 
periodic  safety  inspections  of  airlines  and  aviation-related  activities.3 
These  inspections  include  having  an  inspector  visually  spot-check  an 
airplane  at  the  gate,  monitor  procedures  on  a  scheduled  flight,  or  observe 
maintenance  being  performed  on  an  aircraft  or  its  component  parts.  The 
inspections  cover  four  main  areas: 

Operations  inspections  focus  on  such  items  as  pilots’  certification  and 
performance,  flight  crews’  training,  and  in-flight  record  keeping. 
Maintenance  inspections  examine  an  airline’s  overall  maintenance 
program,  including  the  training  of  aviation  mechanics,  the  development  of 


’Other  primary  functions  Include  certifying  airlines’  operations,  investigating  accidents  and  incidents, 
and  taking  other  steps  to  promote  safety. 
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maintenance  manuals,  and  procedures  for  repairing  aircraft  and  their 
components. 

•  Avionics  inspections  focus  on  electronic  components  of  the  aircraft. 

•  Cabin  safety  inspections  concentrate  on  cabin  procedures,  passenger 
safety,  and  carry-on  baggage. 

In  recent  years,  we  and  others  have  reported  on  problems  with  faa’s 
inspections.4  Reacting  to  these  reports  and  to  a  series  of  fatal  aviation 
accidents  in  the  mid-1990s,  faa  conducted  a  number  of  studies  of  its  own, 
including  the  report  entitled  faa  90  Day  Safety  Review,  issued  in 
September  1996.5  This  review  recommended  that  inspections  be  made 
more  systematic  and  that  they  be  targeted  to  deal  with  identified  risks, 
such  as  airlines’  financial  instability  and  significant  contracting  out  of 
maintenance,  atos,  developed  by  the  Office  of  Flight  Standards  Service 
and  adapted  in  part  from  safety  principles  originally  created  for  the 
nuclear  industiy,  resulted  from  these  recommendations. 


The  ATOS  Concept  Is 
Responsive  to  Many 
Past  Concerns  About 
FAA’s  Safety 
Inspections 


faa  incorporated  features  in  atos  to  address  past  concerns  about 
inspection  quality  and  the  usefulness  of  inspection  data  for  identifying 
potential  safety  threats  and  for  targeting  resources  to  areas  that  pose  the 
greatest  risk.  Before  developing  its  new  aviation  safety  inspection  system, 
faa  analyzed  past  concerns  about  its  inspections.  The  resulting  atos 
inspection  concept  focuses  on  ensuring  that  an  airline  has  operating 
systems  in  place  to  control  the  potential  hazards  and  risks  of  flying  and  to 
prevent  accidents.  FAA  structured  ATOS  to  evaluate  both  an  airline’s 
operating  systems  and  its  adherence  to  those  systems  in  day-to-day 
operations.  In  addition,  faa  incorporated  specific  features  into  atos  to 
make  inspections  more  consistent,  structured,  and  thorough  and  to 
improve  the  collection  and  analysis  of  inspection  data. 


ATOS  Focuses  on  System 
Safety  and  Accident 
Prevention 


faa  emphasizes  a  system  safety  approach  in  atos  that  replaces  routine 
surveillance  and  goes  beyond  spot-checking  airlines  for  compliance  with 
aviation  regulations.  System  safety  involves  the  application  of  technical 
and  managerial  skills  to  identify,  analyze,  assess,  and  control  hazards  and 
risks.  It  covers  every  aspect  of  an  airline’s  operations,  from  the  design  of 


4For  a  list  of  relevant  reports,  see  the  bibliography  and  the  section  citing  related  GAO  products  at  the 
end  of  this  report. 

5 Challenge  2000:  Recommendations  for  Future  Aviation  Safety  Regulation,  prepared  for  the  Federal 
Aviation  Administration  by  Booz-Allen  &  Hamilton  Inc.  (Apr.  1996) ;  and  FAA  90  Day  Safety  Review, 
Federal  Aviation  Administration  (Sept.  16, 1996). 
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the  hardware  to  the  culture  and  attitudes  of  the  airline’s  personnel,  atos 
calls  for  a  systematic  review  of  an  airline’s  policies  and  procedures  to 
ensure  that  they  incorporate  such  basic  safety  principles  as  clear  lines  of 
responsibility  and  written  documentation,  faa  intended  atos  to  ensure  that 
an  airline  has  and  follows  policies  and  procedures  that  build  in  system 
safety,  faa  implemented  atos  on  October  1, 1998,  to  cover  the  nation’s  10 
major  passenger  airlines. 

Under  atos,  faa  assigns  a  team  of  inspectors  to  oversee  each  airline.  Three 
principal  inspectors  lead  the  team,  one  for  each  major  area  of  inspections 
(operations,  maintenance,  and  avionics) .  Additional  team  members  can  be 
based  in  one  of  two  ways.  Inspectors  based  at  the  faa  office  that  holds  the 
airline’s  operating  certificate  work  full  time  on  the  atos  team.  In  contrast, 
field  inspectors,  who  work  in  other  locations  to  which  the  airline  flies, 
work  part  time  on  the  atos  team  and  complete  additional  duties,  such  as 
accident  investigations  and  other  inspections,  for  the  local  faa  office  to 
which  they  are  assigned.  About  540  inspectors  are  assigned  to  the  10  atos 
teams.  Each  atos  team  also  includes  one  cabin  safety  specialist,  whose 
inspections  focus  on  such  areas  as  flight  attendants’  training,  carry-on 
baggage,  and  emergency  evacuation  procedures. 

faa  included  two  kinds  of  guidance  in  atos  to  help  a  team  plan  and  carry 
out  inspections  of  the  airline  it  oversees.  First,  automated  atos  planning 
guidance  is  used  to  develop  the  comprehensive  surveillance  plan  for  each 
airline.  The  planning  guidance  calls  for  using  existing  safety  data,  risk 
indicators,  and  the  inspectors’  knowledge  of  an  airline’s  operations  to 
determine  the  priority  and  frequency  of  inspection  activities.  The  resulting 
comprehensive  surveillance  plan  includes  a  series  of  inspection  tasks  to 
determine  whether  the  airline  has  systems  in  place  to  ensure  safety  and  a 
second  series  of  inspections  to  verify  that  the  airline  is  actually  using  those 
systems,  faa  also  developed  atos  guidance  for  conducting  inspections  that 
is  intended  to  describe  the  tasks  to  be  performed  for  each  type  of 
inspection.  For  a  more  detailed  description  of  the  atos  guidance  and  the 
development  of  the  comprehensive  surveillance  plan,  see  appendix  I. 

faa  designed  atos  to  be  improved  on  an  ongoing  basis,  faa  has  established 
an  atos  Program  Office  to  formulate  and  implement  changes  to  atos  and  to 
support  faa  inspection  teams  through  a  hotline,  help  desk,  and  Web  site. 
FAA  has  also  established  an  internal  audit  team  of  aviation  safety 
inspectors  to  evaluate  the  program,  the  System  Process  Audit  Group.  This 
internal  audit  team  is  an  independent  faA  organization  that  reports  directly 
to  the  Director  of  Flight  Standards. 
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ATOS  Addresses  Many 
Past  Concerns  About 
Inspection  Quality 


In  the  past,  Flight  Standards  typically  allocated  a  large  portion  of  its 
inspection  resources  to  thousands  of  unstructured  inspections  that 
produced  few  reports  of  problems.  We  reported  last  year,  for  example, 
that  inspectors  reported  no  problems  for  96  percent  of  the  inspections 
they  conducted  in  fiscal  years  1990  through  1996.6  Reviews  of  faa’s 
inspection  program  suggest  that  faa  detects  more  problems  through 
rigorous  structured  inspections  than  through  unstructured  inspections. 
Our  recent  review  of  faa’s  oversight  of  the  facilities  airlines  use  to  repair 
aircraft  confirmed  that  standardizing  inspection  tasks  through  the  use  of 
checklists  promotes  more  comprehensive  inspections.7  Past  concerns  also 
included  problems  with  inspectors’  training.  Specifically,  inspectors  have 
performed  inspections  for  which  they  did  not  have  appropriate  or  current 
credentials,  in  part  because  of  limited  funding  for  training.  Providing 
adequate  technical  training  for  faa’s  inspector  workforce  has  proven 
difficult  because  of  the  rapid  change  in  aviation  technology.  In  addition, 
airlines  can  meet  regulatory  requirements  in  a  variety  of  ways,  making  it 
difficult  for  faa’s  inspectors  who  inspect  many  different  airlines  to  be 
familiar  with  the  FAA-approved  procedures  of  each  airline.  Both  principal 
inspectors  and  airline  officials  we  interviewed  said  that  this  lack  of 
familiarity  sometimes  resulted  in  airlines’  being  unfairly  cited  for 
noncompliance.  Finally,  our  review  of  aircraft  repair  facilities  noted  that 
individual  inspectors  generally  identify  far  fewer  deficiencies  than  teams 
do.  The  unstructured  inspection  activities  and  the  underreporting  of 
violations  by  inspectors  resulted  in  inaccurate,  incomplete,  and 
inconsistent  information  that  was  not  very  useful  for  analyzing  safety  risks 
or  targeting  the  agency’s  resources  to  the  problems  that  pose  the  greatest 
risks. 

faa  included  features  in  atos  to  move  toward  more  consistent,  structured 
inspections  by  using  a  system  safety  approach  and  by  providing  new, 
standardized  inspection  tasks,  faa  developed  automated  atos  planning 
guidance  to  ensure  that  inspectors  use  the  same  criteria  to  determine  the 
annual  inspection  activities  for  each  of  the  major  airlines.  An  ATOS  team 
uses  the  planning  guidance  to  identify  potential  problem  areas  at  each 
airline  that  should  be  inspected  more  frequently.  Similarly,  to  standardize 
inspection  activities  across  airlines,  the  guidance  for  conducting 
inspections  lists  tasks  for  each  inspection.  The  inspection  guidance  is 
designed  to  ensure  that  each  inspector  looks  at  an  airline’s  systems  and 


6Aviation  Safety:  Weaknesses  in  Inspection  and  Enforcement  Limit  FAA  in  Identifying  and  Responding 
to  Risks  (GAO/RCED-98-6,  Feb.  27, 1998), 

7Aviation  Safety:  FAA  Oversight  of  Repair  Stations  Needs  Improvement  (GAO/RCED-98-21,  Oct.  24, 
1997]l 
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elements  in  the  same  way.  It  is  also  supposed  to  serve  as  a  checklist  that 
inspectors  can  use  to  ensure  that  their  inspections  are  thorough.  While  the 
old  inspection  system  listed  multiple  tasks  that  could  be  performed, 
inspectors  were  not  required  to  complete  any  specific  tasks  or  to 
document  which  tasks  they  performed.  Thus,  the  scope  of  the  inspection 
work  actually  completed  could  not  be  determined.  In  contrast,  atos 
requires  inspectors  to  document  whether  the  airline  being  inspected 
complied  with  each  item  on  the  inspection  checklist.  The  checklist  also 
serves  as  a  template  for  reporting  inspection  results  in  the  atos  database. 

faa  also  incorporated  team  inspections  in  the  atos  approach.  We  have 
reported  in  the  past  that  teams  have  been  more  effective  than  individuals 
in  identifying  areas  where  airlines  were  not  in  compliance  with  faa 
regulations.  In  many  cases,  the  deficiencies  identified  by  teams  are 
systemic  and  long-standing.  Under  atos,  teams  will  identify  deficiencies 
and  plan  inspections.  Many  inspections  will  be  performed  by  teams  rather 
than  by  individual  inspectors,  as  has  been  done  in  the  past.  Individual 
inspectors  will  continue  to  perform  some  of  the  inspection  work  identified 
in  the  plan. 

FAA  also  included  several  features  in  atos  to  address  past  concerns  about 
inspectors’  training  by  more  effectively  linking  inspectors’  technical 
training  and  qualifications  to  their  job  responsibilities.  First,  faa  designed 
atos  to  link  inspection  assignments  to  the  technical  background  of  each 
inspector  and  to  identify  any  additional  technical  training  needed  to 
accomplish  the  work  plan.  Inspectors  cannot  adequately  inspect  aircraft 
or  systems  unless  they  have  had  the  appropriate  technical  training. 
Second,  inspectors  assigned  to  an  airline,  including  field  inspectors,  must 
complete  training  on  both  atos  and  the  airline’s  specific  policies  and 
procedures  before  they  can  conduct  inspection  activities.  The  training  on 
atos  provides  an  overview  of  the  system  safety  concept  and  how  it  differs 
from  faa’s  past  inspection  approach.  The  training  on  the  airline’s  policies 
and  procedures  familiarizes  inspectors  with  the  approved  operating 
procedures  of  the  airline  they  oversee. 
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ATOS  Addresses  Many 
Past  Problems  With  the 
Usefulness  of  Inspection 
Data 


faa  needs  complete,  accurate  inspection  data  to  target  its  limited 
inspection  resources  to  the  areas  that  pose  the  greatest  potential  safety 
risks.  We  reported  in  1989  and  again  in  1991  on  inaccurate  and  incomplete 
data  in  faa’s  inspection  database  and  recommended  in  1995  that  faa 
develop  a  comprehensive  and  coordinated  strategy  for  improving  its  data.8 
More  recently,  we  reported  last  year  that  70  percent  of  Flight  Standards’ 
inspectors  did  not  enter  all  of  the  violations  they  found  into  their 
inspection  tracking  system  in  fiscal  year  1996.9  Some  inspectors  said  they 
did  not  report  violations  when  compliance  could  be  achieved  informally 
by  bringing  problems  to  the  attention  of  the  airlines.  Others  said  they 
handled  less  serious  violations  informally  because  the  paperwork  involved 
in  reporting  violations  was  too  burdensome,  faa  is  implementing  a 
streamlined  procedure  for  documenting  and  processing  minor 
administrative  violations,  which  should  better  enable  the  agency  to  target 
its  limited  inspection  resources  to  the  areas  that  pose  the  greatest  risks. 
The  streamlined  procedure  will  reduce  paperwork  for  some  types  of 
enforcement  cases,  but  other  efforts  will  be  needed  to  ensure  the 
complete,  accurate  inspection  data  needed  for  improved  targeting  of 
inspection  resources. 

faa  included  several  features  in  atos  to  address  past  concerns  about  the 
usefulness  of  inspection  data  for  analysis  and  targeting.  First,  the 
standardization  of  inspections  and  the  development  of  guidance  for 
planning  and  conducting  inspections  are  steps  intended  to  improve  the 
quality  of  faa’s  data  by  making  inspections  more  systematic  and  thorough. 
When  inspections  are  more  standardized  across  airlines,  data  quality  is 
improved.  Second,  faa  created  a  new  position  within  the  atos  team 
overseeing  each  airline:  a  data  evaluation  program  manager,  whose  job 
will  be  to  review  data  for  validity,  accuracy,  and  completeness  before  they 
are  finalized  in  the  atos  database  for  analysis,  atos  also  added  a  new 
position  for  an  analyst  on  each  team.  The  analyst  is  responsible  for 
collecting  and  analyzing  data  to  support  inspection  planning  and 
retargeting.  Finally,  faa  included  features  in  atos  to  improve  the  targeting 
of  inspection  resources.  FAA  designed  atos  to  allow  the  targeting  of 
inspections  based  on  an  airline’s  size,  operations,  past  history,  and  known 
problem  areas.  The  automated  planning  guidance  can  be  used  to  indicate 
the  risk  factors  applicable  to  the  airline,  such  as  whether  an  airline  is  a 


8 Aviation  Safety:  FAA’s  Inspection  Management  System  Lacks  Adequate  Oversight  (GAO/RCED-90-36, 
Nov.  13, 1989);  Aviation  Safety:  Problems  Persist  in  FAA’s  Inspection  Program  (GAO/RCED-92-14,  Nov. 
20, 1991);  and  Aviation  Safety:  Data  Problems  Threaten  FAA  Strides  on  Safety  Analysis  System 
(GAO/AIMD-95-27,  Feb.  8, 1995). 

9GAO/RCED-98-6,  Feb.  27, 1998. 
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new  entrant  or  contracts  with  other  companies  for  its  maintenance, 
training,  or  ground  handling  services.  Based  on  the  risk  factors,  the 
comprehensive  surveillance  plan  targets  particular  areas,  faa  also 
recognized  that  needed  inspection  work  must  drive  the  assigned 
inspection  resources  for  each  airline.  Finally,  atos  gives  inspectors  the 
flexibility  to  retarget  resources  at  any  point  during  the  year  based  on 
inspection  results.  This  flexibility  allows  faa  to  focus  on  new  problems  as 
they  surface,  rather  than  waiting  until  the  next  year’s  work. 

Because  atos  currently  focuses  on  major  airlines,  it  does  not  address 
concerns  about  the  need  to  provide  additional  oversight  of  new  entrant 
airlines  (that  is,  airlines  in  their  first  5  years  of  operation).  A  separate  faa 
initiative,  the  Certification  Standardization  and  Evaluation  Team,  has 
standardized  and  automated  the  process  for  granting  operating  certificates 
to  new  airlines,  atos  system  safety  concepts  have  been  integrated  into  the 
certification  process  for  new  airlines.  The  new  certification  concept 
includes  a  national  team  to  assist  local  district  offices  in  reviewing  the 
applications  of  new  airlines  and  monitoring  these  airlines  for  their  first  5 
years  of  operation.  As  new  airlines  receive  certification,  faa  plans  to 
oversee  them  using  the  atos  program,  faa  is  not,  however,  currently 
providing  any  additional  oversight  of  new  entrant  airlines  that  were 
already  in  operation  prior  to  the  new  certification  process. 


ATOS’  Design  and 
Implementation 
Problems  Limit  FAA’s 
Efforts  to  Improve 
Safety  Inspections 


Problems  that  emerged  during  design  and  implementation  limit  the 
potential  of  the  atos  concept  to  bring  about  needed  improvements  in  faa’s 
aviation  safety  inspections.  Problems  with  the  atos  inspection  guidance, 
the  links  between  inspectors’  qualifications  and  their  work  assignments, 
and  assembling  effective  teams  affect  the  improvements  envisioned  for 
inspection  quality  and  the  usefulness  of  inspection  data  for  analysis  and 
targeting.  In  addition,  faa  did  not  take  advantage  of  the  expertise  of  airline 
or  industry  representatives  in  developing  atos. 


Design  and 

Implementation  Problems 
Limit  Improvements  to 
Inspection  Quality 


Although  atos  calls  for  (1)  more  systematic,  structured  inspections, 

(2)  closer  links  between  inspectors’  training  and  their  assigned  work 
responsibilities,  and  (3)  greater  use  of  team  inspections  to  improve 
inspection  quality,  its  success  in  the  first  6  months  has  been  limited: 

Inspection  guidance  is  not  complete  and  is  not  sufficiently  clear  and 
detailed  to  accomplish  the  systematic,  structured  inspections  promised  by 
the  atos  concept. 
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ATOS  Inspection  Guidance 
Needs  Significant  Improvement 


atos  has  not  resolved  the  long-standing  problems  with  matching 
inspectors’  qualifications  to  their  work  assignments. 

Team  inspections  are  hampered  by  problems  with  assigning  inspectors, 
including  turnover,  work  locations  that  do  not  match  inspection  needs,  an 
unwillingness  of  inspectors  to  travel,  and  faa’s  inability  to  estimate  the 
resources  needed  to  complete  atos  inspections. 

The  new  inspection  guidance  is  not  adequate  to  ensure  the 
comprehensive,  standardized  inspections  envisioned  by  the  atos  concept. 
We  found  several  problems  with  the  atos  inspection  guidance  as 
implemented. 

Guidance  does  not  cover  all  applicable  regulations.  One  basic  purpose  of 
inspections  is  to  ensure  that  an  airline  complies  with  Federal  Aviation 
Regulations.  Although  atos  aims  to  go  beyond  ensuring  compliance  to  see 
whether  safety  is  built  into  an  airline’s  operations,  compliance  should  also 
be  confirmed.  Principal  inspectors  we  interviewed  expressed  several 
concerns  about  the  link  between  the  atos  inspection  guidance  and  aviation 
regulations.  One  group  of  inspectors  analyzed  the  atos  inspection 
guidance  to  determine  whether  it  covered  all  applicable  Federal  Aviation 
Regulations.  They  identified  296  specific  regulatory  requirements  that  the 
atos  inspection  guidance  did  not  address  out  of  approximately  2,300 
applicable  requirements.  While  we  have  not  verified  each  of  the  296 
requirements  said  to  be  missing,  our  sampling  of  the  inspectors’  results 
confirmed  that  atos  overlooked  some  key  regulatory  requirements.  For 
example,  the  inspection  guidance  does  not  cover  regulations  requiring 
airline  employees  to  be  trained  to  handle  hazardous  materials.  Because  the 
atos  guidance  does  not  include  all  applicable  regulatory  requirements, 
inspections  may  not  be  thorough  enough  to  ensure  compliance. 

Some  guidance  is  not  applicable  to  atos  airlines.  In  addition  to  the 
regulatory  requirements  not  covered  by  atos,  principal  inspectors  we 
interviewed  identified  a  number  of  regulations  referenced  in  atos  that  are 
not  applicable  to  the  major  airlines  currently  under  atos.  For  example,  faa 
requires  that  major  airlines  maintain  an  aircraft’s  weight  and  balance  to 
ensure  that  it  remains  within  approved  limits.  However,  the  ATOS 
inspection  guidance  for  the  weight  and  balance  program  is  based  on  the 
FAA  regulations  governing  commuter  airlines.10  Because  atos  currently 
covers  only  the  largest  airlines,  the  inspection  guidance  should  exclude 


10Commuter  airlines  are  those  that  conduct  scheduled  passenger-carrying  operations  in  aircraft  that 
have  10  or  fewer  seats  and  operate  under  part  135  of  the  Federal  Aviation  Regulations. 
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those  regulations  that  are  applicable  to  commuter  airlines  or  other  types 
of  operators,  such  as  general  aviation  aircraft. 

Principal  inspectors  we  interviewed  also  questioned  the  appropriateness 
of  some  parts  of  the  atos  guidance  that  have  no  basis  in  regulatory 
requirements.  For  example,  the  atos  guidance  calls  for  reviewing  the 
resumes  of  some  airline  officials,  although  regulations  do  not  specify 
qualifications  and  experience  for  their  positions.  Principal  inspectors  also 
questioned  the  basis  for  some  of  the  atos  determinations  that  rely  on  very 
subjective  judgments,  such  as  whether  the  airline  has  a  “safety  focus.” 
They  noted  that  the  atos  guidance  does  not  distinguish  inspection  tasks 
and  findings  based  on  regulations,  which  are  legally  enforceable,  from 
those  based  on  such  other  sources  as  inspector  handbook  guidance,  which 
is  advisory.  Safety  officials  at  most  of  the  major  airlines  echoed  the 
inspectors’  concerns. 

Guidance  is  not  sufficiently  clear  and  detailed.  The  fact  that  atos  provides 
guidance  to  inspectors  on  how  to  plan  and  perform  their  inspections 
represents  a  major  step  toward  the  standardization  of  inspection  tasks. 
However,  the  guidance  it  offers  is  not  yet  thorough  or  detailed  enough  to 
achieve  that  goal.  Principal  inspectors  we  interviewed  questioned  its 
usefulness,  saying  that  it  was  not  clear  or  detailed  enough.  They  reported 
that  they  found  the  language  of  the  guidance  for  planning  inspections 
difficult  to  use  because  it  does  not  detail  the  tasks  to  be  performed  well 
enough.  Staff  at  Sandia  National  Laboratories,  who  were  asked  by  faa  to 
comment  on  the  atos  program  because  of  their  expertise  in  system  safety 
in  the  nuclear  industry,  had  pointed  to  similar  concerns  before  atos  was 
implemented.  In  a  report  on  atos,  the  Sandia  staff  noted  that  the 
inspection  guidance  was  not  based  on  analyses  of  specific  atos  inspection 
tasks  and  the  recording  of  the  results.11  The  Sandia  report  noted  that  a 
proper  task  analysis  describes  the  steps  to  take  and  the  standards  for 
determining  that  the  results  are  correct  and  complete.  The  inspection 
guidance  does  not  provide  this  level  of  detail. 

faa’s  guidance  material  was  not  thoroughly  tested.  The  lack  of  clarity  and 
detail  in  the  atos  guidance,  both  for  (1)  planning  and  (2)  conducting 
inspections,  reflects  the  fact  that  faa  did  not  thoroughly  test  and  validate  it 
before  implementation.  When  FAA  first  tested  the  planning  guidance  in 
April  1998,  the  inspectors  involved  said  they  had  trouble  using  it  because 
the  questions  were  too  vague  and  broad.  When  faa  later  tested  a  revised 


““Analysis  and  Data  Issues  for  the  Implementation  of  a  Systems  Safety  Focus  in  Air  Transportation 
Oversight,”  Oct.  6, 1998. 
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automated  version  of  the  planning  guidance,  most  test  participants  were 
members  of  the  atos  work  group.  Work  group  members  filled  in  the 
answers  without  actually  performing  the  relevant  analysis  or  inspection 
tasks.  While  faa  tested  the  planning  guidance,  it  never  tested  the  guidance 
for  conducting  inspection  tasks. 

Flight  standards  did  not  allow  sufficient  time  for  atos’  implementation. 
While  faa  spent  several  years  developing  the  atos  concept,  the  target 
implementation  date  left  only  7  months  to  take  the  agency’s  new 
inspection  program  from  concept  to  reality.  The  target  date  of  October  1, 
1998,  did  not  allow  time  for  the  adequate  development,  testing,  and 
validation  of  new  inspection  procedures  and  tracking  systems  or  for 
training  and  preparing  the  inspector  workforce.  The  ATOS  work  group 
established  in  February  1998  faced  multiple  challenges  in  meeting  the 
implementation  deadline  in  the  allotted  7  months.  These  challenges 
included  (1)  implementing  a  completely  reengineered  process  for 
conducting  inspections,  (2)  developing  new  automated  systems  for 
planning  atos  inspections  and  tracking  their  results,  and  (3)  preparing  faa 
for  the  cultural  changes  involved  in  having  inspectors  perform  their  work 
differently. 

faa  did  not  adequately  train  inspectors  to  use  atos.  The  full  impact  of  the 
shortcuts  taken  to  meet  the  implementation  deadline  became  apparent 
during  the  inspectors’  training  and,  subsequently,  as  the  teams  of 
inspectors  used  the  new  atos  planning  guidance  to  develop  a 
comprehensive  surveillance  plan  for  each  major  airline  and  to  initiate 
inspections.  FAA  trained  more  than  800  inspectors,  managers,  and  other  faa 
staff  on  the  atos  concept  and  planning  guidance  from  September  through 
November  1998. 

Because  of  the  speed  with  which  the  atos  guidance  and  tracking  systems 
were  developed,  faa  had  little  time  to  develop  training  for  its  inspectors. 
For  example,  very  few  of  the  88  atos  lists  of  inspection  tasks  were 
available  in  time  for  the  training  sessions,  and  the  database  for  reporting 
inspection  results  was  not  yet  completed.  Participants  in  the  training 
sessions  repeatedly  noted  that  the  implementation  of  atos  had  been 
rushed  and  that  there  were  too  many  unknowns  to  allow  for  this  initiative 
to  go  smoothly.  In  course  evaluations  and  in  our  interviews,  inspectors 
said  that  they  were  not  adequately  trained  on  (1)  how  to  perform  the  new 
inspection  tasks,  (2)  how  to  record  the  results  of  their  inspections  in  the 
atos  database,  and  (3)  how  to  use  the  data  on  inspections  to  retarget 
resources.  The  atos  internal  audit  team  concluded,  “The  atos  training  . . . 
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Links  Between  Inspectors’ 
Qualifications  and  Assignments 
Are  Not  Fully  Established 


dedicated  an  insufficient  amount  of  time  to  training  on  the  [inspection 
guidance].”  Standardized  inspections  and  reporting  will  not  occur  without 
clear,  understandable  guidance,  detailed  information  on  tasks  to  be 
completed,  and  training  in  both  conducting  inspections  and  reporting 
findings. 

The  ATOS  model  for  linking  inspectors’  qualifications  and  training  to 
assignments  is  not  yet  a  reality.  We  found  that  some  inspectors  still  lack 
experience  with  faa  or  major  airlines  or  lack  the  specific  training  needed 
to  perform  their  jobs  effectively. 


Lack  of  experience  with  faa  or  major  airlines.  In  selecting  staff  to  fill 
positions  on  ATOS  teams,  faa  did  not  effectively  match  the  qualifications  of 
field  inspectors  to  their  new  positions.  The  selection  of  field  inspectors  to 
fill  atos  assignments  was  not  based  on  any  nationwide  criteria  or 
guidance.  Principal  inspectors  told  us  that  they  had  no  say  in  the  selection 
of  the  field  inspectors  assigned  to  their  teams.  As  a  result,  the 
qualifications  of  some  field  inspectors  did  not  match  the  teams’  needs. 
Several  principal  inspectors  and  managers  said  that  field  inspectors 
assigned  to  them  included  newly  hired  staff  who  were  unfamiliar  with  faa 
or  inspectors — including  experienced  faa  staff — who  lacked  background 
with  major  airlines.  For  example,  one  manager  said  that  some  of  the  field 
inspectors  assigned  to  his  team  had  no  experience  in  large  aircraft  and 
lacked  appropriate  qualifications,  faa  principal  inspectors  have  told  us 
that  it  takes  several  years  to  develop  familiarity  with  the  agency’s 
regulations  and  procedures  or  with  an  airline’s  procedures.  Consequently, 
newly  hired  personnel  may  need  several  years  of  experience  before  they 
can  work  independently  and  be  fully  productive.  Similarly,  inspectors 
trained  on  small  aircraft  may  need  extensive  on-the-job  training  to 
understand  the  workings  of  a  major  airline.  Principal  inspectors 
questioned  whether  some  team  members  had  the  appropriate  skills  to 
oversee  a  major  airline. 


Lack  of  specific  training.  Even  inspectors  who  have  experience  with  major 
airlines  may  not  have  the  specific  technical  qualifications  to  perform  the 
ATOS  inspection  tasks  planned  for  the  airline  to  which  they  are  assigned. 
Principal  inspectors  we  interviewed  said  that  the  technical  qualifications 
of  a  number  of  the  inspectors  assigned  to  them  did  not  match  those 
needed  to  oversee  the  airline  in  question.  For  example,  one  operations 
inspector  said  that  three  of  his  field  inspectors  did  not  have  the 
appropriate  license  required  to  fly  the  aircraft  used  by  the  airline  or  to 
conduct  flight  checks  to  observe  its  pilots.  Similarly,  a  principal  avionics 
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Assignment  Issues  Affect  the 
Quality  of  Team  Inspections 


inspector  said  that  neither  he  nor  his  assistant  had  the  appropriate 
technical  training  in  the  Airbus  aircraft  that  the  airline  flies.  Furthermore, 
the  rapid  changes  in  the  aviation  industry  make  it  difficult  to  provide 
sufficient  technical  training  to  keep  experienced  inspectors  current.  These 
difficulties  in  keeping  pace  with  technological  advances  reflect  faa’s 
persistent  problems  with  providing  technical  training  for  inspectors  and 
matching  inspectors’  qualifications  to  their  job  assignments. 

Staff  turnover,  travel  requirements,  and  conflicting  demands  on  inspectors’ 
time  undermine  faa’s  ability  to  assemble  effective  atos  teams.  Many  of 
these  assignment  issues  cannot  be  resolved  within  the  context  of  the  atos 
program  alone  because  they  involve  broader  decisions  about  inspectors’ 
responsibilities  and  Flight  Standards’  staffing. 

Staff  turnover.  The  turnover  among  field  inspectors  assigned  to  the 
airlines  covered  by  atos  has  made  it  difficult  to  plan  inspection  work  or  to 
meet  the  atos  training  requirements.  Principal  inspectors  said  that  they 
had  lost  field  inspectors  originally  assigned  to  their  teams,  primarily 
because  field  inspectors  went  elsewhere  in  faa  to  accept  promotions  that 
are  not  available  to  them  under  Flight  Standards’  current  staffing 
structure.  For  example,  of  the  28  inspectors  assigned  to  one  atos  team,  11 
had  been  reassigned,  leaving  only  17  available.  Because  some 
replacements  lacked  atos  training,  training  on  the  airline’s  policies  and 
procedures,  or  both,  they  were  ineligible  to  conduct  inspections  under 
atos.  Principal  inspectors  were  concerned  about  how,  if  turnover 
continues,  to  meet  the  requirement  that  inspectors  be  trained  on  the 
policies  and  procedures  of  the  airline  they  oversee.  In  many  cases,  the 
airlines  provided  this  training  in  cooperation  with  faa.  Several  principal 
inspectors  said  that  repeated  requests  for  training  would  be  burdensome 
to  the  airlines.  Principal  inspectors  noted  that  the  turnover  of  field 
inspectors  would  probably  continue  unless  changes  are  made  in  Flight 
Standards’  grade  and  pay  levels  to  permit  field  inspectors  to  be  promoted. 

Inspectors’  work  locations  do  not  always  match  inspection  needs.  Because 
the  work  locations  of  some  inspectors  assigned  to  atos  teams  do  not 
match  inspection  needs,  the  inspectors  will  have  to  travel  to  complete 
their  work.  This  has  made  it  difficult  to  assemble  effective  teams  for 
several  reasons.  First,  many  of  the  principal  inspectors  we  interviewed 
told  us  that  field  inspectors  were  assigned  to  locations  where  they  are  not 
needed,  while  parts  of  the  country  where  the  airlines  have  substantial 
activity  have  no  field  presence.  For  example,  one  principal  inspector  told 
us  that  he  had  a  field  inspector  assigned  to  his  team  who  was  located  in 
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Boise,  Idaho,  where  the  airline  he  oversees  does  not  fly,  but  that  he  had  no 
one  in  Kansas  City,  where  he  needs  an  inspector  to  oversee  maintenance 
operations.  Because  of  these  problems,  a  number  of  principal  inspectors 
suggested  that  they  be  given  a  role  in  identifying  the  needed  qualifications 
and  work  locations  for  field  inspectors  assigned  to  the  atos  teams. 
Furthermore,  in  the  past,  teams  overseeing  an  airline  drew  on  field 
inspectors  in  faa’s  international  offices  to  inspect  foreign  maintenance 
facilities  and  other  overseas  operations  of  airlines  with  international 
routes.  Because  faa  has  not  trained  inspectors  in  its  international  offices 
to  perform  atos  inspections,  atos  team  members  will  have  to  travel 
overseas  to  complete  planned  inspections,  despite  faa’s  already  limited 
travel  funds. 

atos  does  not  resolve  resource  constraints,  faa  designed  atos  to  address 
the  inspection  portion  of  its  inspectors’  workload  and  did  not  address  the 
inspectors’  other  responsibilities.  In  addition  to  inspections,  the  oversight 
of  an  airline  includes  ongoing  activities  referred  to  as  “demand”  work. 
Demand  work  includes  certification  and  approvals  for  initiatives  taken  by 
the  airline,  such  as  adding  new  aircraft  types  to  the  fleet,  adding  new 
destinations,  implementing  computerized  record  keeping,  and 
restructuring  by  management.  The  atos  internal  audit  staff  confirmed  that 
a  conflict  exists  between  demand  work  and  atos  inspection  work. 
Inspectors  based  near  the  offices  of  the  airlines  they  oversee  are  especially 
prone  to  this  conflict  because  they  perform  both  demand  work  and  atos 
inspections.  Many  principal  inspectors  said  that  these  inspectors  do  not 
have  sufficient  time  to  complete  both  their  demand  work  and  atos 
inspection  work.  Field  inspectors,  who  are  assigned  only  part  time  to  atos, 
are  subject  to  different  pressures,  atos  field  inspectors  must  also 
investigate  accidents  and  perform  other  inspection  work  for  the  local 
offices  to  which  they  are  assigned.  Principal  inspectors  expressed  concern 
that  field  inspectors  might  not  be  available  for  ATOS  work  when  needed 
because  of  demands  from  their  local  offices.  In  addition,  several  principal 
inspectors  and  managers  questioned  whether  the  emphasis  on  atos  has 
shifted  resources  away  from  other  areas  of  concern,  such  as  repair 
stations,  troubled  smaller  carriers,  and  general  aviation  safety.  Managers 
contended  that  such  conflicts  between  demand  work  and  inspection 
activities  will  persist  as  long  as  atos  focuses  solely  on  inspections,  rather 
than  having  a  broader  view  of  the  work  that  inspectors  perform. 

These  conflicting  demands  on  inspectors’  time  and  on  resources  such  as 
travel  funds  are  exacerbated  because  managers  do  not  know  what 
resources  will  be  needed  to  perform  atos  inspections.  The  atos  guidance 
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The  ATOS  Data  Collection 
Process  Limits  the 
Usefulness  of  Data  for 
Analysis  and  Targeting 


FAA  Did  Not  Adequately 
Determine  Data  Analysis  Needs 


does  not  include  estimates  of  how  long  various  inspection  tasks  should 
take.  Without  such  information,  it  is  difficult  to  estimate  how  many 
inspectors  will  be  needed  to  perform  the  tasks  or  what  travel  funds  they 
may  need  to  accomplish  the  planned  work.  Both  inspectors  and  their 
managers  told  us  that  they  are  unable  to  estimate  how  much  time  many  of 
the  atos  inspection  tasks  will  require.  Two  principal  inspectors  told  us  that 
managers  of  some  local  offices  will  not  approve  atos  work  plans  for  field 
inspectors  unless  they  know  how  much  time  will  be  needed  to  complete 
the  work.  Testing  and  validation  of  the  atos  guidance  would  have  provided 
preliminary  estimates  for  the  time  needed  to  complete  various  atos 
inspection  tasks. 


Although  atos  includes  initiatives  to  address  past  problems  with  the 
usefulness  of  faa’s  inspection  data  for  analysis  and  targeting  of  resources 
to  the  greatest  safety  risks,  this  goal  may  not  be  fully  realized  for  several 
reasons.  First,  because  of  the  problems  with  the  ATOS  guidance  that  we 
have  already  discussed,  the  data  collected  from  atos  inspections  are  not 
likely  to  be  reliable  enough  to  support  meaningful  analyses.  Furthermore, 
in  translating  atos  from  concept  to  design  and  eventual  implementation, 
faa  did  not  adequately  determine  its  data  analysis  needs.  The  volume  of 
inspection  data  available  for  analysis  has  also  dropped  dramatically  under 
atos  because  few  inspections  have  been  completed.  Finally,  principal 
inspectors  responsible  for  overseeing  airlines  did  not  have  timely  access 
even  to  the  limited  data  available  until  faa  granted  them  access  in 
May  1999. 

While  atos  calls  for  structured  inspections  intended  to  result  in  more 
thorough  and  consistent  data,  the  way  atos  collects  data  limits  the 
potential  of  its  database  as  a  tool  for  analysis  and  targeting.  The  data 
limitations  reflect  the  fact  that  faa  did  not  sufficiently  analyze  its  data 
needs  before  developing  atos  inspection  guidance  and  its  automated 
database.  The  development  of  an  effective  automated  system  begins  with  a 
thorough  analysis  of  the  data  required  to  meet  the  needs  of  those  using  the 
database.  In  the  case  of  atos,  a  thorough  analysis  would  describe  in  detail 
the  questions  that  need  to  be  asked  to  improve  aviation  safety,  determine 
precisely  what  data  are  needed  to  answer  those  questions,  and  plan  the 
appropriate  analyses  to  be  conducted  on  those  data  to  answer  the 
questions.  After  the  preliminary  analysis  of  the  users’  data  needs  is 
completed,  the  prototype  system  must  be  tested,  validated,  and  revised  in 
an  iterative  process  between  data  users  and  automation  developers.  Staff 
from  Sandia  National  Laboratories  reviewed  ATOS  and  concluded  that  its 
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developers  did  not  go  beyond  abstract,  high-level  statements  about  users’ 
needs  to  the  level  of  detail  essential  to  ensure  data  quality.  They  added 
that  without  more  detailed  information  it  will  be  difficult  to  identify  the 
data  and  information  needed  to  answer  questions  about  aviation  safety 
and  impossible  to  support  the  detailed  automated  analysis  of  an  airline. 

Because  faa  did  not  sufficiently  analyze  or  list  the  data  needed  by 
inspectors  to  make  determinations  about  aviation  safety  and  did  not 
adequately  test  or  validate  atos,  features  that  would  maximize  the 
usefulness  of  data  for  analysis  and  targeting  were  not  built  into  the  system. 
We  found  four  specific  limitations  with  the  atos  database. 

Key  information  is  not  required,  atos  does  not  currently  require  inspectors 
to  record  inspection  data  that  are  essential  for  effective  analyses.  To 
perform  effective  analyses  of  safety  data,  basic  information  is  needed, 
such  as  the  airline’s  name;  the  make,  model,  and  series  of  the  aircraft;  the 
aircraft  and  pilot  identification  numbers;  and  where  the  inspection  was 
performed.  However,  an  atos  inspection  record  can  be  closed  without  any 
of  this  basic  information  having  been  entered  because  the  system  requires 
only  that  inspectors  indicate  whether  an  airline  is  in  compliance  and 
explain  any  violations.  For  example,  our  review  of  completed  atos 
inspections  found  18  completed  inspection  activities  related  to  de-icing 
operations  for  which  the  location  had  been  entered,  not  in  the  appropriate 
field,  but  in  a  comment  field.  Data  contained  in  comment  fields  cannot  be 
used  for  automated  analyses  to  determine,  for  example,  which  airports  are 
experiencing  problems  with  de-icing. 

In  addition,  the  atos  inspection  guidance  does  not  provide  inspectors  with 
an  index  or  clear  instructions  on  where  in  the  atos  database  to  report 
findings.  If  an  inspector  cannot  easily  figure  out  where  to  report  a  finding, 
it  may  go  unreported.  For  example,  an  inspector  who  found  incorrect 
safety  placards  on  an  aircraft  said  he  had  to  search  atos  for  nearly  4  hours 
to  record  this  violation.  Inspectors’  underreporting  of  violations  that  they 
observed  has  jeopardized  the  completeness  and  quality  of  inspection  data 
in  the  past,  atos  does  not  resolve  the  problems  that  contributed  to 
underreporting,  and  the  difficulty  in  finding  where  to  report  violations  in 
atos  may  exacerbate  this  problem. 

Response  options  preclude  meaningful  analyses.  Many  atos  inspection 
activities  are  to  be  summed  up  in  a  report  with  a  single  “yes”  or  “no” 
response  to  the  items  on  the  inspection  checklist  for  a  given  area.  If  an 
inspector  finds  problems  during  a  single  inspection  activity,  it  results  in  a 
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“no”  for  the  entire  question.  Regardless  of  whether  an  inspector  finds  one 
problem  in  10  inspections  or  one  in  100,  a  “no”  results.  Without  knowing 
the  proportion  of  “no”  responses  for  a  given  item,  faa  cannot  use  the 
inspection  results  to  conduct  quantitative  analyses  on  atos  data,  for 
example,  a  trend  analysis  to  determine  whether  the  airline’s  compliance  in 
a  given  area  has  improved. 

Minimum  number  of  inspection  activities  needed  is  not  specified,  atos 
currently  does  not  specify  the  minimum  number  of  activities  needed  to 
complete  each  inspection.  Instead,  individual  inspectors  decide 
independently  how  many  times  inspection  observations  should  be 
performed  to  determine  whether  the  airline  follows  its  procedures  and 
complies  with  regulations.  This  determination  is  subjective.  In  our  review 
of  the  atos  data  available  as  of  March  31, 1999,  we  found  that  four 
inspectors  conducted  anywhere  from  2  to  12  observations  to  complete  the 
record  for  the  same  type  of  inspection.  Because  these  reports  result  in 
“yes”  or  “no”  answers  on  the  completed  inspection  report,  rather  than  in 
a  quantifiable  report  of  the  proportion  of  “no”  answers,  the  data 
submitted  by  these  inspectors  cannot  be  compared  or  analyzed.  Because 
inspectors  will  never  have  enough  time  to  observe  every  safety-related 
component  of  an  airline’s  system,  it  is  important  to  define  the  minimum 
number  of  inspections  to  be  conducted  and  to  report  accurately  the 
proportion  of  instances  of  noncompliance. 

atos  does  not  link  to  Flight  Standards’  existing  data  analysis  system.  The 
information  in  the  atos  database  cannot  be  analyzed  by  Flight  Standards’ 
existing  aviation  risk  analysis  system,  the  Safety  Performance  Analysis 
System  (spas)  .  faa  has  spent  $95  million  developing  spas  to  analyze  key 
aviation  safety  data,  identify  trends  and  potential  safety  concerns,  and 
target  inspection  resources  accordingly.  In  January  1998,  the  atos 
development  team  noted  that  atos  inspection  data  could  not  be  analyzed 
by  spas.  The  development  team  recommended  that  atos  not  be 
implemented  until  faa  could  develop  appropriate  links  between  atos  and 
spas.  However,  faa,  in  its  efforts  to  meet  the  October  1, 1998, 
implementation  date,  went  forward  without  addressing  this 
recommendation,  spas  program  officials  told  us  that  significant  work 
remains  to  link  the  two  systems.  An  ongoing  faa  work  group  studying 
ways  to  incorporate  system  safety  into  Flight  Standards’  programs  has 
been  given  responsibility  for  linking  atos  and  spas.  This  work  group  has 
not  yet  established  a  timetable  for  how  or  when  the  link  will  be  completed. 
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Availability  of  ATOS  Data 
Limits  Analysis 


The  usefulness  of  atos  data  for  analysis  and  targeting  has  been  limited  by 
the  overall  lack  of  information  in  the  atos  database  as  well  as  by  initial 
problems  with  accessing  what  information  is  available,  faa  devoted  much 
of  the  early  part  of  fiscal  year  1999  to  planning  ATOS  inspections  and 
training  inspectors  in  the  new  system.  As  a  result,  very  few  data  on  the 
performance  of  the  airlines  covered  by  atos  have  been  available  for 
analysis.  By  the  end  of  the  second  quarter,  fewer  than  60  of  approximately 
5,400  planned  inspections  had  been  completed,  entered  into  the  atos 
database,  and  made  available  to  principal  inspectors.  These  completed 
inspections  comprised  fewer  than  1,200  individual  inspection  activities, 
substantially  fewer  than  those  recorded  in  previous  years.12  The  flow  of 
inspection  activity  results  into  the  atos  database  has  since  accelerated, 
with  an  additional  267  inspections  and  1,808  individual  inspection 
activities  recorded  over  the  first  6  weeks  of  the  third  quarter. 

Nevertheless,  only  326  inspections,  consisting  of  3,079  inspection 
activities,  had  been  completed  by  May  11, 1999,  and  for  one  airline  no 
inspections  had  been  completed.  In  contrast,  during  the  first  two  quarters 
of  fiscal  year  1998,  over  37,000  inspection  activities  had  been  recorded  in 
the  Program  Reporting  and  Tracking  System  for  the  10  airlines  now 
covered  by  atos.13 

We  also  found  that  access  to  the  information  available  in  the  ATOS  database 
was  limited  until  inspections  were  completed,  faa  designed  atos  so  that 
neither  principal  inspectors  nor  analysts  could  access  data  until  inspectors 
completed  all  inspection  activities  and  the  team’s  data  evaluation  program 
manager  reviewed  and  approved  the  data.  The  lack  of  access  to  key  safety 
data  created  problems  for  principal  inspectors,  who  are  responsible  for 
overseeing  operations,  maintenance,  and  avionics  inspections  at  each 
airline.  They  could  not  view  inspection  results  until  the  data  evaluation 
program  manager  reviewed  the  data  for  clarity  and  consistency.  Several 
principal  inspectors  expressed  concern  about  not  having  timely  access  to 
key  safety  data,  which  they  use  as  an  early  warning  of  potential  safety 
risks.  In  February  1999,  we  briefed  faa  on  the  concerns  expressed  about 
delayed  access  to  inspection  results  in  the  atos  database.  In  March  1999, 
faa  directed  its  automation  contractor  to  take  the  steps  necessary  to  grant 
principal  inspectors  immediate  read-only  access  to  inspection  findings. 
Until  this  access  was  granted,  principal  inspectors  remained  dependent  on 


12Under  ATOS,  each  inspection  recorded  in  the  database  summarizes  the  results  of  multiple 
observations  called  inspection  activities. 

13While  the  3,079  ATOS  inspection  activities  may  not  be  directly  comparable  to  the  37,000  inspections 
conducted  under  the  previous  inspection  system,  a  substantial  drop-off  in  the  inspection  information 
available  to  managers  and  analysts  has  clearly  occurred  over  the  first  half  of  fiscal  year  1999. 
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such  informal  means  as  e-mail  and  telephone  calls  to  learn  of  any 
problems  or  violations  found  during  inspections.  Principal  inspectors  now 
have  access  to  all  final  inspection  activity  reports  related  to  their  specialty 
for  the  airline  that  they  oversee.  This  change  made  data  on  2,724  atos 
inspection  activities  available  to  principal  inspectors  as  of  May  11, 1999. 

ATOS’  Targeting 

Capabilities  Are  Limited  by 
Inadequate  Inspection 
Guidance  and  Problems 

With  the  Usefulness  of 

Data 

The  previously  mentioned  inadequacies  in  the  atos  guidance  and  problems 
with  the  usefulness  of  the  resulting  data  for  analysis  limit  faa’s  ability  to 
use  atos  for  targeting  inspection  resources  to  the  problems  that  pose  the 
greatest  safety  risks.  Prior  to  the  implementation  of  atos,  the  primary 
purpose  of  inspections  was  to  identify  individual  safety  problems  and 
ensure  their  correction.  Under  atos,  the  primary  objective  is  to  provide 
reliable  data  to  enable  FAA  to  identify  the  highest-priority  safety  concerns 
and  target  the  agency’s  resources  to  reduce  these  concerns  or  risks  as  well 
as  to  ensure  that  individual  safety  problems  are  corrected.  As 
implemented,  atos  falls  short  of  this  goal.  Significant  revisions  will  be 
needed  to  the  atos  guidance  and  database  before  the  potential  of  this  new 
inspection  system  can  be  realized. 

FAA  Did  Not  Take 

Advantage  of  Industry 
Expertise  in  Developing 
ATOS 

Although  FAA  has  many  ongoing  initiatives  with  the  aviation  industry, 
coordination  with  industry  was  lacking  in  the  design  of  atos.  Although 
several  aspects  of  atos  involve  areas  in  which  industry  safety  experts  have 
experience  and  similar  goals,  the  design  process  did  not  include  airline  or 
industry  representatives,  who  were  briefed  on  atos  after  the  concept  was 
developed.  The  industry’s  input  is  missing,  for  instance,  in  the  risk  weights 
atos  uses  in  its  planning  guidance  to  help  determine  the  number  of 
inspections  a  team  is  to  conduct,  thus  directing  resources  to  areas  that 
require  additional  oversight.  For  example,  the  weight  atos  gives  to  an 
airline’s  screening,  boarding,  and  briefing  procedures  for  passengers  is 
greater  than  that  assigned  to  cockpit  procedures.  FAA  neither  validated 
these  weights  nor  discussed  them  with  airline  safety  officials. 

FAA  Has  Initiated 
Improvements  to 

ATOS 

We  briefed  FAA  in  December  1998  and  again  in  February  1999  on  the 
problems  we  had  found  with  the  implementation  of  atos.  Because  of  the 
magnitude  and  seriousness  of  the  problems  associated  with  the  current 
atos  guidance  and  database,  we  suggested  that  faa  not  expand  atos  to 
additional  airlines,  repair  stations,  or  other  aviation  operations  until  these 
problems  are  resolved.  Acknowledging  that  there  were  significant 
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challenges  to  further  implementing  the  program,  faa  agreed  that 
expansion  of  the  program  needs  to  be  delayed.14 

In  March  1999,  the  Director  of  Flight  Standards  and  key  atos  program 
officials  met  with  the  principal  inspectors  who  lead  the  10  atos  teams  to 
brief  them  on  our  findings  and  to  obtain  their  views  on  the  system’s 
implementation.  This  meeting  resulted  in  a  list  of  immediate  and  future 
actions  to  be  undertaken  to  address  the  concerns  raised.  First,  faa  has 
taken  steps  to  provide  principal  inspectors  with  immediate  access  to 
completed  inspection  reports.  FAA  also  provided  partial  funding  for 
planned  overseas  inspections,  which  will  allow  inspectors  to  complete 
some  of  the  more  critical  overseas  inspections.  Finally,  faa  will 
incorporate  additional  information  on  the  inspection  guidance  task  lists 
and  database  tracking  system  into  atos  training  beginning  in  July.  These 
initiatives  will  help  alleviate  several  of  the  more  immediate  problems  with 
atos  by  (1)  making  key  safety  information  available  to  principal  inspectors 
as  soon  as  inspections  are  completed,  (2)  allowing  at  least  some  of  the 
planned  overseas  inspections  to  take  place,  and  (3)  providing  better 
training  for  inspectors  on  how  to  conduct  inspections  and  record  their 
results.  In  addition,  both  the  atos  Program  Office  and  the  atos  internal 
audit  group  will  continue  to  monitor  the  implementation  closely  to  ensure 
that  these  problems  and  others  that  may  arise  are  addressed.  Funding  has 
not  yet  been  allocated  to  support  the  needed  improvements  to  atos  or  to 
link  atos  with  faa’s  existing  data  targeting  and  analysis  system,  spas. 


Conclusions 


The  atos  concept  offers  significant  promise  for  helping  faa  overcome 
deficiencies  in  its  past  approach  to  aviation  safety  inspections.  However, 
its  potential  will  not  be  fully  realized  until  faa  resolves  the  problems 
resulting  from  the  ambitious  schedule  it  followed  in  implementing  atos. 
These  problems  limit  both  faa’s  ability  to  conduct  more  systematic, 
structured  inspections  and  analyze  the  resulting  data  to  identify  safety 
trends  and  its  ability  to  target  its  resources  to  the  greatest  risks.  The  atos 
guidance  is  not  clear  and  detailed  enough  to  ensure  more  systematic, 
structured  inspections  that  will  result  in  more  usable  data.  In  addition,  faa 
has  not  adequately  analyzed  the  data  needs  of  atos  users  to  ensure  that  the 
system  collects  the  information  that  will  enable  the  agency  to  perform 
critical  trend  and  safety  analyses.  Such  analyses  are  also  limited  because 
atos  does  not  link  to  faa’s  other  major  database  for  safety  analyses  (spas), 
faa  has  recognized  the  need  for  significant  improvements  before  atos  will 


MFAA  already  had  plans  to  bring  two  additional  airlines  into  ATOS  in  the  near  future.  These  airlines 
have  recently  completed  certification  activities  and  will  begin  ATOS  inspections  upon  completion  of  a 
transition  phase. 
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Recommendations 


achieve  its  full  potential  as  a  system  for  overseeing  commercial  airline 
safety,  and  the  agency  has  taken  actions  to  correct  some  of  the  problems 
we  found.  Because  of  the  challenges  involved  in  making  the  needed 
improvements  to  atos,  faa  officials  have  postponed  plans  to  expand  atos 
to  other  airlines  until  these  problems  are  corrected.  In  addition,  faa  is 
aware  that  the  resolution  of  some  problems  with  the  implementation  of 
atos  involve  broader  issues  that  concern  staffing  decisions  and  workload 
issues  that  can  affect  all  of  Flight  Standards’  inspectors. 


To  strengthen  the  efforts  to  improve  faa’s  aviation  safety  inspections  and 
the  usefulness  of  the  data  that  result  from  these  inspections  for  analysis 
and  for  targeting  the  agency’s  resources  to  the  greatest  potential  safety 
threats,  we  recommend  that  the  Secretary  of  Transportation  direct  the  faa 
Administrator  to  take  the  following  actions: 

•  Develop  a  structured  process  and  timeline  for  working  with  inspectors  to 
revise  the  Air  Transportation  Oversight  System’s  planning  and  inspection 
guidance.  The  process  should  involve  the  inspectors  now  using  this 
guidance  to  (1)  identify  problems  with  the  clarity  of  the  guidance, 

(2)  revise  the  inspection  guidance  to  include  tasks  related  to  all  applicable 
Federal  Aviation  Regulations,  and  (3)  define  the  tasks  to  be  completed 
during  inspections. 

•  Revise  the  inspection  guidance  to  (1)  include  guidelines  on  the  minimum 
number  of  times  to  perform  various  inspection  tasks  and  (2)  distinguish 
between  tasks  based  on  regulatory  requirements  and  those  based  on 
handbook  or  other  guidance. 

•  Develop  a  plan  that  involves  both  inspectors  and  experts  in  risk 
assessment  and  database  development  in  revising  and  refining  the  analysis 
of  the  data  needs  of  users  of  the  new  inspection  program.  The 
requirements  analysis  should  describe  in  detail  the  questions  that  need  to 
be  asked  to  improve  safety,  determine  precisely  what  data  are  needed  to 
answer  those  questions,  and  plan  the  appropriate  analyses  to  be 
conducted  on  those  data  to  answer  the  questions. 

•  Restructure  the  inspection  database  to  (1)  require  that  essential  data  fields 
be  completed  before  inspection  reports  can  be  closed  out  and  (2)  clearly 
indicate  the  proportion  of  inspection  observations  in  which  an  airline 
complies  with  regulations. 
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•  Determine  what  revisions  will  be  needed  to  the  Air  Transportation 
Oversight  System  database  and  the  agency’s  existing  Safety  Performance 
Analysis  System  database  to  maximize  the  potential  of  these  two  systems 
by  coordinating  their  trend  analyses  to  identify  potential  safety  risks. 

•  Test  and  validate  the  revised  guidance  and  database  for  the  new 
inspection  program. 


Agency  Comments 
and  Our  Evaluation 


We  provided  faa  with  a  draft  of  this  report  for  review  and  comment.  We 
met  with  the  Deputy  Associate  Administrator  for  Regulation  and 
Certification,  the  Director  of  the  Office  of  Flight  Standards  Service,  the 
Acting  Manager  of  the  Air  Transportation  Oversight  System  Program 
Office,  the  Manager  of  the  System  Process  Audit  Group,  and  other  faa 
officials.  The  agency  agreed  with  the  substance  of  the  report  but 
commented  that  the  tone  was  unnecessarily  negative  and  could  leave  the 
impression  that  we  believe  that  the  program  should  be  abandoned.  Agency 
officials  also  said  that  they  made  a  conscious  decision  to  implement  the 
new  system  aggressively,  rather  than  in  stages,  and  recognized  that  this 
approach  would  result  in  some  implementation  problems.  However,  they 
believe  that  ultimately  the  new  system  will  be  fully  operational  sooner 
than  if  they  followed  a  more  conservative  implementation  approach.  The 
agency  also  commented  that  our  review  of  the  program  was  premature 
and  suggested  that  an  evaluation  of  the  program  in  another  year  would 
find  that  most  of  the  problems  we  reported  had  been  resolved. 

We  do  not  believe  the  program  should  be  abandoned.  We  believe  that  our 
report  clearly  supports  the  Air  Transportation  Oversight  System  and 
acknowledges  its  potential  for  significantly  strengthening  faa’s  inspection 
process.  However,  we  continue  to  believe  that  serious  challenges  need  to 
be  overcome  before  this  program  can  achieve  its  potential.  Because  our 
review  coincided  with  the  program’s  implementation,  we  were  able  to 
identify  serious  problems  early  and  to  promote  constructive  action  by  faa 
to  begin  resolving  them.  As  we  reported,  FAA  has  begun  to  address  some  of 
these  problems.  In  commenting  on  this  report,  faa  also  provided  some 
updated  information  on  its  inspection  activities  and  suggested  wording 
revisions  that  we  incorporated  as  appropriate. 


We  conducted  our  work  from  September  1998  through  June  1999  in 
accordance  with  generally  accepted  government  auditing  standards. 
Appendix  II  contains  details  of  the  scope  and  methodology  of  our  review. 
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As  you  requested,  unless  you  publicly  announce  its  contents  earlier,  we 
plan  no  further  distribution  of  this  report  until  10  days  from  the  date  of 
this  letter.  We  will  then  send  copies  to  the  appropriate  congressional 
committees;  Rodney  E.  Slater,  the  Secretary  of  Transportation;  Jane  F. 
Garvey,  the  Administrator,  faa;  Jacob  J.  Lew,  the  Director,  Office  of 
Management  and  Budget;  and  other  interested  parties.  We  will  also  make 
copies  available  to  others  upon  request. 

If  you  have  any  questions  about  this  report  or  need  additional  information, 
please  call  me  at  (202)  512-2834.  Major  contributors  to  this  report  are 
listed  in  appendix  III. 


John  H.  Anderson,  Jr. 

Director,  Transportation  Issues 
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The  Air  Transportation  Oversight  System  (atos)  is  a  process  designed  to 
improve  the  Federal  Aviation  Administration’s  (faa)  oversight  of  airlines. 
faa’s  Office  of  Flight  Standards  Service  developed  atos  with  the  support  of 
Sandia  National  Laboratories,  atos  uses  system  safety  principles  created 
for  the  nuclear  industry  and  risk  management  to  ensure  that  airlines  have 
safety  built  into  their  operating  systems.  A  systems  safety  approach  means 
that  faa’s  inspection  efforts  will  cover  all  aspects  of  an  airline’s 
performance  that  can  affect  safety  and  will  focus  on  preventing  accidents. 

A  certificate  management  team  oversees  each  of  the  10  airlines  under 
atos.  Each  team  is  led  by  three  principal  inspectors,  one  for  each  major 
area  of  inspections  (operations,  maintenance,  and  avionics).  Additional 
team  members  include  those  based  at  the  faa  office  that  holds  the  airline’s 
operating  certificate  and  field  inspectors  in  other  faa  offices  at  locations 
to  which  the  airline  flies. 

The  team  uses  automated  planning  guidance  to  develop  a  comprehensive 
surveillance  plan  for  the  airline.  The  planning  guidance  consists  of  two 
automated  tools — the  system  safely  analysis  tool  (ssat)  and  the  air  carrier 
assessment  tool  (acat).  The  principal  inspectors  complete  the  ssat  and 
acat  prior  to  an  annual  planning  meeting.  During  the  meeting,  team 
members  discuss  the  ssat  and  acat,  and  their  feedback  is  included  in  the 
final  version.  The  results  of  the  acat  help  define  inspection  activities  that 
the  team  will  include  in  the  airline’s  comprehensive  surveillance  plan.  The 
ssat,  acat,  and  comprehensive  surveillance  plan  are  described  in  more 
detail  below.  Figure  1.1  shows  how  these  atos  components  relate  to  one 
another. 


Page  28 


GAO/RCED-99-183  FAA’s  New  Aviation  Inspection  System 


Appendix  I 

ATOS  Guidance  and  the  Comprehensive 
Surveillance  Plan 


Figure  1.1:  How  FAA  Designs  Comprehensive  Surveillance  Plans  Through  ATOS 
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Source:  Federal  Aviation  Administration. 
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Using  the  automated  planning  guidance,  the  team  analyzes  the  airline’s 
operations,  which  are  divided  into  seven  systems  (see  table  1.1).  These 
systems  are  made  up  of  14  subsystems  and  of  79  elements.  For  example, 
the  flight  operations  system  is  made  up  of  two  subsystems,  which 
encompass  elements  such  as  aircraft  dispatch  and  flight  deck  procedures. 
faa  designed  the  ssat  to  help  the  team  assess  the  systems  in  place  at  an 
airline  to  ensure  safe  operations.  The  acat  applies  a  set  of  risk  indicators 
to  the  airline’s  subsystems  and  elements  to  generate  a  comprehensive 
surveillance  plan.  These  risk  indicators  for  the  acat  are  based  on  safety 
and  performance  information  that  reflects  areas  of  potential  risk  for  an 
airline’s  operations.  Hence,  the  comprehensive  surveillance  plan  will 
target  those  areas  most  likely  to  have  safety  problems. 


Table  1.1:  The  Seven  Airline  Systems 
Defined  in  ATOS 


Manuals 

Controls  the  information  and  instructions 
that  define  and  govern  an  airline's 
activities. 

Flight  operations 

Governs  aircraft  movement. 

Personnel  training  and  qualifications 

Ensures  that  an  airline's  personnel  are 
trained  and  qualified. 

Route  structures 

Maintains  an  airline's  facilities  on  approved 
routes. 

Airman/crew  member  flight,  rest,  and  duty 
time 

Prescribes  time  limitations  for  airline 
employees. 

Technical  administration 

Addresses  all  other  aspects  of  an  airline's 
certification  and  operations. 

Source:  FAA  Order  8400.10,  Air  Transportation  Operations  Inspector's  Handbook,  appendix  6. 


System  _  System’s  purpose _ 

Aircraft  configuration  and  control  Maintains  the  physical  condition  of  the 

aircraft  and  associated  components. 


The  System  Safety 
Assessment  Tool 


The  ssat  is  a  computerized  tool  designed  to  focus  the  inspection  team’s 
attention  on  the  systems  that  an  airline  has  in  place.  It  poses  questions  to 
the  team  covering  six  categories:  safety  attributes,  safety  culture, 
communications,  accountability,  training  programs,  and  potential  problem 
areas.  (See  table  1.2.)  The  principal  inspectors  complete  the  ssat  prior  to  a 
yearly  meeting  to  plan  inspections  to  oversee  the  airline’s  operations.  To 
complete  the  ssat,  the  principal  inspectors  rely  on  their  knowledge  of  the 
airline  and  on  the  data  available  through  faa’s  Safety  Performance 
Analysis  System  (spas),  the  Flight  Standards  Automated  System,  or  other 
sources.  Before  the  annual  meeting,  the  team  members  review  the  ssat 
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Table  1.2:  Categories  of  System  Safety 
Used  With  SSAT 


completed  by  the  principal  inspectors  and  provide  feedback.  The  SSAT  is 
finalized  at  the  annual  meeting  but  may  be  revised  during  the  year  to 
retarget  inspection  resources. 


Category 

Definition 

Safety  attributes 

ATOS  identifies  six  safety  attributes: 
Responsibility.  The  unit  or  person  in  the 
airline  that  determines  the  course  of  action 
for  a  process. 

Authority:  The  unit  or  person  in  the  airline 
that  has  the  authority  to  establish  or  modify 
a  process. 

Procedures:  A  documented  method  of 
accomplishing  a  process. 

Controls:  A  check  or  restraint  that  is 
designed  into  a  process  to  ensure  a 
desired  result. 

Process  measurements:  The  unit  or  person 
in  the  airline  that  measures  and  assesses 
information  to  identify,  detect,  analyze,  and 
document  problems  or  potential  problems. 
Interfaces:  Points  at  which  independent 
processes  interact. 

Safety  culture 

The  priority  given  to  safety  by  the  airline's 
systems,  including  the  airline's 
identification  and  response  to  safety  risks, 
and  the  effectiveness  of  internal  evaluation 
systems. 

Communications 

The  communication  and  feedback 
channels  within  the  airline  to  report  and 
respond  to  safety  risks  as  well  as  open 
and  timely  communication  with  FAA  and 
equipment  manufacturers. 

Accountability 

The  extent  to  which  the  airline  holds  its 
management  and  employees  accountable 
for  their  assigned  responsibility  and 
authority. 

Training  programs 

The  priority  an  airline  places  on  training  as 
well  as  the  effectiveness  of  initial  and 
recurrent  training  programs. 

Potential  problem  areas 

The  existence  of  concerns  based  on 
previous  accidents  or  incidents,  hotline 
complaints,  or  trends  revealed  in  safety 
data. 

Source:  FAA  Order  8400.10,  Air  Transportation  Operations  Inspector's  Handbook,  appendix  6. 
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By  completing  the  SSAT,  the  ATOS  inspection  team  assesses  how  well  an 
airline  addresses  system  safety  issues.  Using  this  information,  the  team 
determines  whether  to  inspect  any  of  the  systems  more  or  less  frequently 
than  suggested  by  the  atos  guidance  and  incorporates  the  inspections  in 
the  comprehensive  surveillance  plan  for  the  airline. 


The  Air  Carrier 
Assessment  Tool 


In  completing  the  acat,  principal  inspectors  use  the  results  of  the  ssat, 
their  knowledge  of  the  airline  their  team  oversees,  and  any  other  available 
data  to  indicate  concerns  about  any  real  or  potential  problem  that  could 
contribute  to  the  failure  of  one  of  the  airline’s  elements,  subsystems,  or 
systems.  The  acat  applies  risk  indicators  to  each  of  the  airline’s  systems. 
Table  1.3  shows  the  types  of  risk  indicators  that  are  assessed  when 
inspectors  complete  an  acat. 


Table  1.3:  ACAT  Risk  Indicators 


Type  of  risk  indicator 

Definition  and  examples 

Operational  stability 

Those  aspects  of  an  airline's  organization 
and  environment  over  which  it  has  no 
direct  control  and  that,  when  managed 
effectively,  could  enhance  system  safety 
and  stability  (e.g.,  turnover  in  personnel,  or 
a  merger  or  takeover). 

Airline  dynamics 

Aspects  of  an  airline's  environment  that  it 
directly  controls  and  that  could  be  used  to 
enhance  system  safety  and  stability  (e.g., 
an  internal  evaluation  program,  and  risk 
management). 

Performance  history 

The  results  of  an  airline's  operations  over 
time  (e.g.,  enforcement  actions, 
self-disclosure  reports  to  FAA). 

Environmental  criticality 

Those  aspects  of  an  airline's  surroundings 
that  could  lead  to  or  trigger  a  failure  of  one 
of  its  systems,  subsystems,  or  elements 
and  potentially  create  an  unsafe  condition 
(e.g.,  age  of  the  fleet,  outsourcing  of 
maintenance). 

Source:  FAA  Order  8400.10,  Air  Transportation  Operations  Inspector's  Handbook,  appendix  6. 

The  principal  inspectors  complete  the  acat  prior  to  the  annual  planning 
meeting.  During  the  annual  planning  meeting,  the  team  members  provide 
feedback  on  the  acat  that  is  included  in  the  final  version. 
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The  Comprehensive 
Surveillance  Plan 


The  comprehensive  surveillance  plan  is  automatically  generated  based  on 
the  information  the  team  enters  into  acat.  This  provides  a  baseline 
surveillance  plan  that  is  tailored  to  the  airline,  reflecting  concerns 
indicated  by  the  principal  inspectors.  Each  comprehensive  surveillance 
plan  incorporates  two  types  of  inspections,  safety  attribute  inspections 
(sai)  and  element  performance  inspections  (epi).  sais  appraise  the  quality 
of  an  airline’s  safety  attributes  (see  table  1.2)  for  each  system,  its 
subsystems,  and  its  elements.  A  team  of  inspectors  conducts  these  system 
inspections,  epis  determine  whether  an  airline  adheres  to  its  written 
procedures  and  controls  for  each  system  element  and  whether  the 
established  performance  measures  for  each  element  are  met.  Individual 
inspectors  conduct  these  inspections. 

atos  allows  the  principal  inspector  to  increase  or,  in  some  cases,  decrease 
the  level  of  inspection  generated  by  the  surveillance  plan.  This  allows 
principal  inspectors  to  use  their  expertise  and  personal  knowledge  of  the 
airline  to  target  resources  toward  the  greatest  safety  risks. 

Although  the  comprehensive  surveillance  plan  is  automatically  generated 
based  on  the  results  of  the  ssat  and  acat,  the  plan  is  not  finalized  until  the 
annual  inspection  planning  meeting,  which  is  attended  by  all  members  of 
the  team.  This  allows  the  principal  inspectors  to  discuss  the  completed 
ssat  and  acat  and  to  make  changes  based  on  other  inspectors’  feedback. 
In  addition,  work  assignments  are  discussed  and  made  for  each  of  the  sais 
and  epis  that  are  planned.  Principal  inspectors  complete  and  approve  the 
final  plan. 


Safety  Attribute 
Inspections 


The  acat  provides  information  directly  to  the  sai  planning  system  in  atos, 
which  indicates  an  inspection  priority  for  each  of  the  airline’s  subsystems. 
Considering  the  sai  priority,  a  principal  inspector  enters  the  number  of  sais 
to  be  completed  for  each  of  the  airline’s  elements  during  the  year. 
Automation  of  the  sai  also  allows  a  principal  inspector  to  assign  teams  for 
each  of  the  SAI  activities.  The  principal  inspector  can  also  provide  specific 
instructions  to  the  team  regarding  the  inspections. 

An  sai  is  an  in-depth  look  at  an  airline’s  policies  and  procedures  for  a 
system  element.  This  inspection  is  structured  to  look  at  the  safety 
attributes  shown  in  table  1.2.  An  SAI  is  completed  by  a  team  of  inspectors, 
led  by  a  team  coordinator.  This  team  assesses  the  accuracy  and 
completeness  of  written  policies  and  procedures  governing  each  safety 
attribute  associated  with  one  of  the  airline  system  elements.  For  example, 
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one  element  of  the  route  structure  system  is  the  line  stations  for  servicing 
and  maintaining  the  airline’s  aircraft  at  each  city  it  serves.  Teams 
performing  line  station  sais  determine  if  there  are  qualified  persons 
accountable  for  the  line  stations  and  if  those  persons  have  the  authority  to 
change  the  processes  governing  those  facilities.  The  teams  also  review  the 
procedures  governing  line  stations  to  determine  if  proper  controls  are  in 
place — such  as  standards  for  the  maintenance  conducted  at  the  line 
station  or  for  the  training  of  line  station  employees.  Finally,  the  teams 
determine  if  processes  are  in  place  to  identify  and  correct  problems  as 
well  as  to  ensure  that  other  processes,  such  as  de-icing  and  refueling,  are 
coordinated. 


Element  Performance 
Inspections 


faa  established  a  baseline  for  how  frequently  each  of  the  79  atos  elements 
should  be  inspected.  This  frequency  baseline  identifies  whether  a  system 
element  should  be  inspected  on  an  annual,  semiannual,  or  quarterly  basis 
within  the  planning  cycle.  The  ACAT  calculates  an  assessment  value  that, 
when  applied  to  the  frequency  baseline,  increases  or  decreases  the 
number  of  inspections  based  on  the  concerns  the  principal  inspectors 
have  identified.  Elements  must  be  inspected  at  least  once  a  year.  Once  the 
principal  inspectors  have  determined  the  number  of  epis  that  will  be 
conducted,  the  work  is  assigned  to  other  inspectors  on  the  team  using  the 
automated  system. 


An  epi  shows  whether  the  airline  follows  the  airline’s  procedures  and 
controls.  Individual  inspectors  conduct  epis,  which  most  resemble  the 
routine  inspections  faa  conducted  in  the  past.  For  example,  the  line 
station  epi  requires  an  inspector  to  visit  a  line  station  to  determine  if 
procedures  and  controls  in  place  are  being  followed  at  that  location.  A  line 
station  inspection  under  ATOS  may  include  multiple  visits  to  one  location 
or  to  a  variety  of  other  locations  as  well.  All  of  these  visits  may  be  included 
in  a  single  epi  report. 
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In  September  1998,  the  Chairman  and  Ranking  Democratic  Member  of  the 
Subcommittee  on  Aviation,  House  Committee  on  Transportation  and 
Infrastructure,  asked  us  to  address  questions  related  to  faa’s  new  Air 
Transportation  Oversight  System: 

•  To  what  extent  does  atos  address  past  concerns  about  faa’s  aviation 
safety  inspections? 

•  What  factors,  if  any,  surfaced  during  the  implementation  of  atos  that  could 
impede  its  success? 

•  What  is  faa  doing  to  address  any  factors  that  could  impede  the  success  of 

ATOS? 

To  determine  to  what  extent  atos  addresses  problems  identified  in  the 
past  with  faa’s  inspection  program,  we  reviewed  previous  reports  by  gao, 
the  Department  of  Transportation’s  Inspector  General,  and  internal  faa 
reports,  such  as  faa  90  Day  Safety  Review.  In  addition,  we  attended  atos 
training  provided  to  faa  inspectors  and  the  annual  inspection  planning 
meeting  held  by  1  of  faa’s  10  certificate  management  teams.  We 
interviewed  members  of  the  atos  work  group  and  program  office  to 
discuss  how  the  new  program  was  developed  and  the  agency’s  intentions 
for  its  implementation.  In  total,  we  interviewed  68  faa  employees  assigned 
to  the  airlines  atos  covers,  including  64  of  540  atos  inspectors  and  4  of  the 
10  unit  supervisors.  The  64  inspectors  included  28  of  the  30  principal 
inspectors  who  oversee  atos  airlines.  In  the  two  cases,  we  interviewed  the 
assistant  principal  inspectors  because  the  principal  inspectors  were  not 
available.  We  discussed  the  atos  concept,  training,  and  implementation 
with  each  inspector.  In  addition  to  the  68  FAA  employees  assigned  to 
oversee  airlines  under  atos,  we  interviewed  five  Flight  Standards  district 
office  managers  and  supervisors  who  oversee  atos  field  inspectors  to  gain 
a  broader  perspective  on  inspector  workload  issues  beyond  those 
involving  atos  inspections.  We  also  reviewed  all  10  comprehensive 
surveillance  plans  developed  for  the  10  airlines  covered  by  atos,  as  well  as 
inspection  findings  reported  through  May  11, 1999. 

To  determine  what  factors,  if  any,  surfaced  during  the  implementation  of 
atos  that  could  impede  its  success,  we  interviewed  faa’s  principal 
inspectors  for  all  10  airlines  covered  by  atos  and  also  inspectors 
(including  field  inspectors)  assigned  to  the  certificate  management  teams. 
We  interviewed  staff  from  Sandia  National  Laboratories  who  served  as 
consultants  on  the  atos  project  and  reviewed  Sandia’s  reports  on  the 


Page  35 


GAO/RCED-99183  FAA’s  New  Aviation  Inspection  System 


Appendix  II 

Objectives,  Scope,  and  Methodology 


system’s  implementation.  At  faa  headquarters,  we  interviewed  data 
management  and  system  officials  as  well  as  staff  assigned  to  the  System 
Process  Audit  Group.  We  also  reviewed  reports  prepared  by  these  faa 
units  on  the  implementation  of  atos.  In  addition,  we  discussed  atos 
specifics  with  key  safety  officials  at  the  10  airlines  covered  by  the  system. 

To  determine  what  faa  is  doing  to  address  any  factors  that  could  impede 
the  success  of  atos,  we  reviewed  internal  reports  on  atos  and  associated 
recommendations.  We  discussed  these  recommendations  and  what  impact 
faa’s  budget  shortfall  will  have  on  atos  inspection  efforts  with  principal 
inspectors  on  the  atos  teams  and  with  Flight  Standards  headquarters 
officials. 

We  conducted  our  work  from  September  1998  through  June  1999  in 
accordance  with  generally  accepted  government  auditing  standards. 
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